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Executive  Summary 


The  Institute  for  Defense  Analyses  (IDA)  was  asked  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD(AT&L))  to  address  the  question:  What  are  ways  of  assessing  acquisition  time  and  issues  of  acquisition  cycle  time  and 
cycle  time  growth ?  In  considering  this  question,  IDA  placed  it  in  the  following  context:  How  much  should  the  Department  of  Defense 
(DOD)  focus  on  managing  cycle  time  or  schedule  per  se?  Or  is  cycle  time  generally  driven  by — a  function  of — other  decisions  that  if 
managed  properly  would  result  in  reasonable  cycle  times  from  the  standpoint  of  providing  capabilities  when  needed?  What  is  the 
basis  for  determining  what  is  reasonable? 

Overall  Findings 

1.  Acquisition  program  cycle  times  are  slightly  longer  now  than  in  past  decades. 1  However,  there  is  little  evidence  that  this  is, 
of  itself,  a  problem,  or  what  problems  increasing  cycle  times  either  cause  or  indicate.  Nonetheless,  DOD  program 
development  approaches  result,  in  the  aggregate,  in  too  many  programs  simultaneously  chasing  too  few  dollars,  such  that 
the  chance  of  all  programs  being  effectively  implemented  as  scheduled  is  unlikely.  There  is  clear  evidence  that  stretching 
programs  results  in  increased  costs  overall  and  per  unit  acquired.  There  is  high  variability  in  program  development  time. 
This  raises  the  question:  Are  there  management  measures  for  disciplining  the  weapons  development  process  that  can 
overcome  these  dynamics? 

2.  Low  priority  and  focus  given  to  setting  initial  schedules  is  a  management  issue  that  the  USD(AT&L)  might  want  to 
address.  These  initial  schedules  become  embedded  in  contracts  and  affect  subsequent  acquisition  milestones,  but  the 
processes  that  determine  these  schedules  appear  to  be  analytically  weak.  What  can  be  done  to  improve  schedule 
development?  How  does  the  requirements  process  address  schedules  and  how  are  requirement  imperatives  translated  into 
program  schedules?  This  would  include  addressing  how  the  requirements  process  develops  and  provides  inputs  into 


This  statement  is  based  on  the  Performance  of  the  Defense  Acquisition  System,  2013  Annual  Report,  which  states  that  “development  cycle  time  on  contracts 
after  1980  took  an  average  of  0.9  years  longer  than  contracts  before  1980. . .”  Office  of  the  Secretary  of  Defense,  Performance  of  the  Defense  Acquisition 
System,  2013  Annual  Report  (Washington,  DC:  DOD,  28  June  2013),  55. 


program  schedules.  The  Office  of  the  Secretary  of  Defense  (OSD)  might  want  to  take  on  this  issue,  but  doing  so  would 
require  addressing  underlying  factors  in  the  development  process. 

3.  Long  cycle  times  frequently  result  from  the  pursuit  of  highly  ambitious  technical  capabilities  combined  with  a  program 
management  framework  lacking  appropriate  mechanisms  for  identifying  and  reducing  technical  risk.  Are  there 
development  management  approaches  (such  as  prototyping)  that  would  better  identify  technical  risks  and  discipline 
developments  to  accommodate  risk?  What  is  needed  to  apply  such  measures  effectively? 

4.  Developmental  shortfalls  not  addressed  early  will  stretch  cycle  time.  When  initial  operational  test  and  evaluation  (IOT&E) 
reveals  problems  late  in  the  process,  development  schedules  are  often  stretched.  This  raises  fundamental  questions:  Are  the 
developmental  testing  (DT)  and  early  operational  testing  (OT)  inadequate?  Are  the  reporting  and  decision  processes 
ineffective?  Are  there  actions  OSD  could  take  to  improve  the  rigor  of  DT  and  early  OT,  or  to  better  address  the  schedule 
impacts  from  problems  identified  in  OT? 

5.  Efforts  to  dramatically  shorten  cycle  times  appear  to  be  episodic  and  short-lived.  Despite  periodic  calls  for  dramatically 
reduced  cycle  times  (such  as  a  50  percent  reduction),  various  rapid  development  processes  employed  during  times  of 
conflict  are  seen  as  exceptions  and  fall  into  disuse  when  the  conflict  ends. 

Three  Different  Management  Problems 

The  assessment  identified  three  different  management  problems  related  to  acquisition  cycle  time. 

1 .  Setting  realistic  schedules:  The  first  problem  is  effectively  assessing  what  a  reasonable  schedule  for  a  program  should  be 
relative  to  explicit  choices  of  performance  capabilities:  How  much  of  what  is  needed  by  when?  How  time  constrained  is 
the  need  relative  to  the  level  of  capabilities  needed?  Analyses  have  shown  that  DOD  programs  have  been  overly  optimistic 
in  determining  how  long  it  will  take  to  achieve  the  stated  capabilities.  Moreover,  there  appears  to  be  limited  analytical 
focus  on  assessing  the  time  it  will  take  to  achieve  one  level  of  capabilities  relative  to  another  level  and  trading  off  the  level 
of  performance  in  favor  of  earlier  delivery.  Another  factor  in  setting  realistic  schedules  is  identifying  and  providing  for 
uncertainties  in  achieving  the  desired  outcomes.  Should  programs  fund  parallel  risk  reduction  efforts,  such  as  the 
development  of  alternative  designs  for  key  components,  with  the  expectation  that  this  will  reduce  the  potential  for  delays? 
Producing  realistic  and  well-determined  initial  schedules  appears  to  be  a  fundamental  problem  for  defense  management. 

2.  Reducing  schedule  growth:  Once  a  schedule  is  set,  data  show  that  programs  often  do  not  adhere  to  the  schedule,  do  not 
meet  intermediate  milestone  dates,  and  usually  deliver  late.  We  believe  this  stems  largely  from  the  first  problem:  overly 


IV 


optimistic  initial  schedules  cannot  be  met  and  thus  they  will  likely  “creep”  longer.  However,  even  with  well-formulated 
initial  schedules,  incentives  and  exigencies  tend  toward  longer  times  than  planned.  Thus,  a  useful  focus  would  be 
identifying  motivations,  incentives,  and  other  factors  that  foster  schedule  growth  and  addressing  how  these  can  be 
mitigated. 

3.  Reducing  cycle  times:  Data  show  that  DOD  major  systems  development  tend  to  take  many  years — on  the  average  of  seven 
to  ten  years.  Moreover  over  recent  decades,  the  cycle  time  has  gotten  longer.  It  is  important  to  note  that  DOD  systems 
stress  several  domains,  such  as  performance,  operational  deployment  environment,  and  length  of  system  use,  that  are 
extreme  compared  to  most  commercial  systems.  Achieving  shorter  schedules  for  highly  complex  and  technically 
challenging  systems  must  be  accomplished  with  careful  regard  to  the  overall  outcome;  often  trying  to  do  too  much  too  soon 
has  resulted  in  unsatisfying  outcomes  and  often  longer,  not  shorter,  development  times. 

This  leads  to  the  observation  that  properly  designing  both  the  system  and  program  are  essential.  For  a  projected  schedule  to  be 
meaningful  and  useful,  the  project  itself  must  be  well-formed,  based  on  sound  engineering,  and  a  schedule  derived  from  the  work 
needed  to  achieve  the  intended  result.  No  schedule  can  make  up  for  a  faulty  design. 

From  this  overall  perspective  five  focus  questions  were  identified  for  further  research: 

1 .  Priority  and  Focus :  How  important  is  cycle  time  in  the  decision  process  and  how  do  the  current  processes  for  setting  and 
assessing  schedules  reflect  the  management  interest? 

2.  Requirements  Definition:  How  are  requirements  for  programs  set  and  how  do  these  impact  program  schedules?  Related  to 
this  is  the  question  of  what  type  of  information  is  available  and  what  analyses  are  done  during  requirements  formulation 
that  impact  on  setting  the  initial  schedules?  When  can  realistic  program  schedules  be  set  relative  to  the  degree  of  program 
definition? 

3.  Management  and  Oversight :  How  are  schedules  set,  assessed,  and  evaluated  relative  to  overall  management  of  the 
program? 

4.  Program  Definition  and  Characteristics :  How  do  types  of  programs — different  types  of  systems  or  capabilities — affect 
how  long  a  program  takes  and  how  well  a  program’s  schedule  can  be  predicted  and  maintained? 

5.  Approaches  to  Reduce  Cycle  Time :  What  has  been  tried?  What  has  worked?  When  is  substantially  reduced  time-to-product 
appropriate? 
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The  following  diagram  portrays  how  these  assessment  areas  relate  to  the  three  key  management  problems  stated  above. 


As  this  research  elaborates,  some  of  these  assessment  areas  seem  of  greater  importance  to  particular  management  problems,  but 
this  is  also  something  that  will  be  better  understood  as  research  proceeds.  The  large  Xs  in  the  slide  denote  the  areas  hypothesized  to  be 
of  greatest  relevance  for  future  research  focused  on  that  management  problem — for  example  in  “setting  schedules,”  “priority”  and 
“requirements”  are  likely  to  be  of  greatest  relevance,  while  “management  and  oversight”  is  less  of  a  factor  (denoted  by  a  smaller  x). 
Alternatively,  for  “schedule  growth”  our  hypothesis  is  that  “management  and  oversight”  will  be  the  dominant  factor. 

“Program  characteristics”  are  depicted  as  an  additional  dimension,  as  the  impact  of  different  types  of  systems  or  program 
characteristics  on  setting  schedules  and  managing  them  is  seen  as  variable  and  largely  undetermined — that  is,  certain  types  of  systems 
may  pose  much  larger  or  different  challenges  than  other  types.  However,  it  is  not  yet  clear  from  available  empirical  analyses  which 
programs  have  which  effects.  Thus,  while  program  characteristics  may  be  an  important  factor  in  reducing  cycle  times,  specifically 
which  types  of  programs  would  have  such  effects  is  an  open  empirical  question. 

Overall,  this  matrix  indicates  that  more  research  is  needed  in  the  areas  identified  to  lead  to  useful  management  interventions  for 
reducing  weapon  system  acquisition  cycle  time. 
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Research  Focus 


How  much  should  the  Department  of  Defense  (DOD)  focus,  specifically,  on  managing  program  development  cycle  time 
(schedule)?  Or  is  cycle  time  principally  driven  by — a  function  of — other  decisions  that  if  managed  properly  would  result  in 
“reasonable”  cycle  times  that  provide  capabilities  when  needed?  What  is  the  basis  for  determining  what  is  reasonable? 

The  Institute  for  Defense  Analyses’  (IDA)  perspective,  based  on  its  research  and  a  review  of  the  broader  literature,  is  that  cycle 
time  is  largely  determined  by  a  set  of  decisions  in  three  domains: 

1 .  How  well  the  program  is  formulated  at  its  inception  relative  to  what  is  known  and  taken  into  consideration  about  the 
program  risks  related  to  the  technologies  to  be  developed  and  incorporated 

2.  The  clarity,  specificity,  and  stability  of  the  user  requirement  for  the  capability 

3.  The  priority  of  the  program  relative  to  adequate  initial  funding  and  persistent  funding  during  budget  cycles 

This  analysis  draws  upon  prior  IDA  research,  as  well  as  studies  by  the  RAND  Corporation,  the  Military  Services  (primarily  the 
U.S.  Air  Force),  the  Office  of  the  Secretary  of  Defense  (OSD)  and  others.  IDA  also  reviewed  data  analyses  that  have  been  conducted 
for  these  studies  and  identified  additional  data  and  assessments  that  may  provide  greater  understanding  of  program  cycle  time.  The 
objective  of  this  initial  investigation  of  cycle  time  issues  was  to  define  a  program  of  future  research  that  would  illuminate  the  causes 
of  problematic  growth  in  acquisition  program  cycle  times  and  identify  management  mechanisms  to  address  those  causes. 
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ida  |  Research  Focus 

•  Research  question:  What  are  ways  of  assessing  acquisition 
time  and  issues  of  acquisition  “cycle  time”  and  “cycle  time 
growth”  that  provide  useful/actionable  inputs  for  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD(AT&L))? 

•  Approach:  First  phase  is  a  “Capstone  Perspective” 

-  Addresses  cycle  time  issue  from  a  management  perspective. 

-  What  does  the  data  show  are  prospects  for  understanding  cycle  time 
and  managing  its  effects? 

-  Define  set  of  hypotheses  and  analyses  on  specific  types  of  causes  to 
help  identify  management  “levers”  or  approaches. 

-  Based  on  data  and  a  literature  review  to  identify  alternative 
hypotheses  demonstrating  management  implications  for  further 
assessment. 

•  Deliverable:  A  research  program  defining  the  analytical 
tasks  to  address  acquisition  cycle  time  as  a  management 
problem 
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Summary  of  Cycle  Time  Issues 


Acquisition  program  cycle  times  are  slightly  longer  now  than  in  past  decades. 1  There  is  little  evidence  that  this  is,  of  itself,  a 
problem,  or  what  problems  increasing  cycle  times  either  cause  or  indicate.  There  is  clear  evidence  that  stretching  programs  results  in 
increased  costs  overall  and  per  unit  acquired.  There  is  high  variability  in  program  development  times.  Are  there  management  measures 
for  disciplining  the  weapons  development  process  that  can  overcome  these  dynamics? 

Low  priority  and  focus  given  to  setting  initial  schedules  is  a  management  issue  that  the  USD(AT&L)  might  want  to  address. 
These  initial  schedules  become  embedded  in  contracts  and  affect  subsequent  acquisition  milestones,  but  the  processes  that  determine 
these  schedules  appear  to  be  analytically  weak.  What  can  be  done  to  improve  schedule  development?  How  does  the  requirements 
process  address  schedules  and  how  are  requirement  imperatives  translated  into  program  schedules? 

Long  cycle  times  frequently  result  from  the  pursuit  of  highly  ambitious  technical  capabilities  combined  with  a  program 
management  framework  lacking  appropriate  mechanisms  for  identifying  and  reducing  technical  risk.  Are  there  development 
management  approaches  (such  as  prototyping)  that  would  better  identify  technical  risks  and  discipline  developments  to  accommodate 
risk?  What  is  needed  to  apply  such  measures  effectively? 

Developmental  shortfalls  not  addressed  early  will  stretch  cycle  time.  When  initial  operational  test  and  evaluation  (IOT&E) 
reveals  problems  late  in  the  process,  development  schedules  are  often  stretched.  Are  the  developmental  testing  (DT)  and  early 
operational  testing  (OT)  adequate?  Are  the  reporting  and  decision  processes  effective?  Are  there  actions  OSD  could  take  to  improve 
the  rigor  of  DT  and  early  OT,  or  to  better  address  the  schedule  impacts  from  problems  identified  in  OT? 

Efforts  to  dramatically  shorten  cycle  times  appear  to  be  episodic  and  short-lived.  Despite  periodic  calls  for  dramatically  reduced 
cycle  times  (such  as  a  50  percent  reduction),  various  rapid  development  processes  employed  during  times  of  conflict  are  seen  as 
exceptions  and  fall  into  disuse  when  the  conflict  ends. 


This  statement  is  based  on  the  Performance  of  the  Defense  Acquisition  System,  2013  Annual  Report,  which  states  that  “development  cycle  time  on  contracts 
after  1980  took  an  average  of  0.9  years  longer  than  contracts  before  1980. . .”  Office  of  the  Secretary  of  Defense,  Performance  of  the  Defense  Acquisition 
System,  2013  Annual  Report  (Washington,  DC:  DOD,  28  June  2013),  55. 
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IDA  Summary  of  Cycle  Time  Issues 

•  Cycle  times  are  seen  as  too  long  and  are  getting  longer 

-  Is  the  future  years  defense  program  (FYDP)-“packed”  with  development 
programs  that  cannot  be  executed  on  the  promised  schedules  with  the 
allocated  funding? 

•  Planning  for  and  overseeing  acquisition  schedules  has  been 
given  relatively  low  priority  and  focus 

•  Extremely  challenging  and  changing  requirements  lead  to 
“beyond -state-of-the-art”  technological  approaches  that  are  not 
adequately  developed  within  a  planned  program  schedule  and 
allocated  funding 

•  The  “front-end”  of  the  development  process,  while  crucial  for 
setting  achievable  acquisition  cycle  times,  often  results  in  poorly 
defined  programs  with  optimistic  schedules  and  cost  estimates 

•  Established  processes  and  review  mechanisms  for  disciplining 
schedule  often  appear  to  be  ineffective  or  ignored 

•  Little  progress  has  been  made  in  shortening  cycle  times 
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Framing  Perspective 


The  approach  taken  in  this  front-end  analysis  was  to  look  broadly  across  the  entire  domain  of  defense  program  development  and 
acquisition  to  identify  key  factors,  trends,  and  contributors  to  program  cycle  times  with  an  emphasis  on  factors  that  can  be  addressed 
by  management  practices.  At  the  outset  it  is  important  to  emphasize  that  the  goal  of  program  development  and  acquisition  is  getting 
needed  new  capabilities  in  the  form  of  new  or  improved  weapon  systems  to  military  users.  These  are  the  requirements  that  drive  the 
development  and  acquisition  process.  Related  to  the  development  and  acquisition  of  such  capabilities  are  considerations  of  when  such 
capabilities  are  needed  and  the  degree  to  which  such  capabilities  are  worth  their  likely  cost.  It  is  well  recognized  that  these  three 
elements — performance,  schedule,  and  cost — are  closely  interrelated  and  should  be  managed  in  a  coherent  manner  that  understands 
and  evaluates  these  relationships.  Another  aspect  of  defense  system  development  and  acquisition  is  that  many  of  these  systems  are 
large  and  complex.  Military  systems  are  technically  demanding  compared  to  most  other  types  of  products,  such  as  consumer  products, 
and  often  are  projected  to  be  in  use  for  decades.  This  scale,  complexity,  and  length  of  use  makes  the  assessment  of  defense  systems 
development  and  acquisition  extremely  demanding  and  creates  risks  in  achieving  the  desired  performance,  identifying  costs  in 
advance,  and  determining  and  executing  realistic  schedules.  Simply  put,  as  DOD  seeks  demanding  capabilities  relative  to  identified  or 
projected  threats,  or  to  take  advantage  of  emerging  complex  advanced  technologies,  the  job  of  projecting  when  the  capability  can  be 
developed  and  acquired,  as  well  as  what  it  will  cost  will  be  more  difficult  than  for  many  commercial  systems.  The  focus  of  this 
research  is  on  ways  to  improve  DOD’s  acquisition  management  process. 
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IDA  Framing  Perspective 

Objective:  Produce  an  exploratory  assessment  that  is 

•  Encompassing — spans  the  problem 

•  Illuminating — contributes  to  understanding 

•  Practical — focuses  on  management  concerns 

This  is  the  “front-end  of  the  front-end” — 

Review  and  synthesize  literature  and  assessments  of  defense 
acquisition,  other  government  processes,  and  commercial 
industry  with  emphasis  on  identifying  factors  that  contribute  to 
systems  development  times  and  processes  for  [1]  managing 
and  [2]  [possibly]  reducing  cycle  times. 

•  Reviewed  over  60  documents — articles,  research  studies, 
dissertations,  theses,  government  documents,  etc. 

•  Reviewed  recent  data  analyses  conducted  by  OSD 

•  Interviewed  former  government  officials,  industry  execs 

•  Held  internal  roundtables  with  IDA  subject  matter  experts  (SME) 
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Three  Different  Management  Problems 


IDA  has  identified  three  different  management  problems  related  to  acquisition  cycle  time: 

1.  Setting  realistic  schedules:  The  first  problem  is  effectively  assessing  what  a  reasonable  schedule  for  a  program  should  be 
relative  to  the  chosen  performance  capabilities.  The  fundamental  question  is  when  is  the  capability  needed?  How  time 
constrained  is  the  need  relative  to  the  level  of  capabilities  needed?  For  a  projected  schedule  to  be  useful,  the  project  itself 
must  be  well-formed,  based  on  sound  engineering  and  the  schedule  derived  from  the  work  needed  to  achieve  the  intended 
result.  Analyses  have  shown  that  DOD  programs  have  generally  been  overly  optimistic  about  determining  how  long  it  will 
take  to  achieve  the  stated  capabilities  and  a  realistic  assessment  of  technology  readiness.  Moreover,  there  appears  to  be 
limited  analytical  focus  on  assessing  the  time  it  will  take  to  achieve  one  level  of  capabilities  relative  to  another  level  and 
trading  off  the  level  of  capabilities  in  favor  of  earlier  delivery.  Another  factor  in  setting  realistic  schedules  is  identifying 
and  providing  for  uncertainties  and  risks  in  achieving  the  desired  outcomes.  Do  programs  fund  parallel  risk  reduction 
efforts,  such  as  the  development  of  alternative  designs  for  key  components,  with  the  expectation  that  this  will  reduce  the 
potential  for  delays?  This  problem  of  realistic  and  well-determined  initial  schedules  is  a  fundamental  question  for  defense 
management. 

2.  Reducing  schedule  growth:  Once  a  schedule  is  set,  data  show  that  programs  often  do  not  adhere  to  the  schedule — the 
programs  usually  do  not  meet  intermediate  milestone  dates  and  usually  deliver  late.  IDA  beleves  that  this  stems  largely 
from  the  first  problem:  overly  optimistic  initial  schedules  cannot  be  met  and  thus  they  will  likely  “creep”  longer.  However, 
even  with  well-formulated  initial  schedules,  incentives  and  exigencies  tend  to  push  time  to  results  longer  than  planned, 
especially  if  no  contingencies  are  provided  for  uncertainty.  Thus,  identifying  incentives  that  foster  schedule  growth  and 
addressing  how  these  can  be  mitigated  should  be  a  useful  focus. 

3.  Reducing  cycle  times:  Data  show  that  major  defense  systems  development  and  acquisition  tend  to  take  many  years — on  the 
average  of  seven  to  ten  years.  Some  have  argued  that  this  is  too  long — often  using  commercial  industry  as  a  basis  for 
comparison.  It  is  important  to  note  that  DOD  systems  stress  several  domains  such  as  performance,  operational  deployment 
environment,  and  length  of  system  use,  that  are  extreme  compared  to  most  commercial  systems.  Moreover,  it  is  not  clear 
that  commercial  systems  that  have  similarly  stressing  characteristics  are  developed  and  delivered  in  substantially  less  time. 
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^  Three  Different  Management  Problems 

1)  Setting  realistic  schedules 

•  Explicit  analysis  in  the  schedule/performance  requirements  trade-space 
leading  to  a  decision  before  the  schedule  is  set  and  codified  in  an 
approved  program  plan 

•  Establishment  of  executable  schedules  for  a  range  of  potential  program 
performance  goals 

2)  Reducing  schedule  growth 

•  Enforcement  of  realistic  schedules 

•  Stable  funding  (with  adequate  reserves?) 

•  Authority  to  trade  performance  goals  for  schedule — Who?  Program 
Manager  (PM)? 

3)  Substantially  reducing  cycle  times 

•  Developing  and  implementing  new  acquisition  strategies  and  attendant 
processes  for  more  iterative,  adaptive  development  and  acquisition 

Proper  upfront  design  of  the  weapon  system  and 
the  program  are  crucial  to  getting  schedules  right 
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Focus  Areas  for  Understanding  Cycle  Time 


IDA’s  review  has  identified  the  following  questions  as  key  for  assessing  program  cycle  time  from  a  management  perspective: 

1.  Priority  and  Focus — How  important  is  cycle  time  in  the  decision  process  and  how  do  the  current  processes  for  setting  and 
assessing  schedules  reflect  the  management  interest? 

2.  Requirements  Definition — How  are  program  requirements  set  and  how  do  these  impact  program  schedules?  A  related 
question  is  what  type  of  information  is  available  and  considered  during  requirements  formulation  that  impact  on  setting  the 
initial  schedules?  There  is  also  the  underlying  question  of  when  realistic  program  schedules  can  be  set  relative  to  the 
degree  of  program  definition?  One  hypothesis  is  that  program  schedule  dates,  such  as  initial  operational  capability  (IOC) 
are  set  too  early  in  the  program  definition  process,  become  “hardwired”  into  the  program  (particularly  if  reported  to 
Congress),  and  these  become  unrealistic  as  the  program  is  subsequently  pursued. 

3.  Management  and  Oversight — How  are  schedules  set,  assessed  and  evaluated  relative  to  overall  management  of  the 
program? 

4.  Program  Definition  and  Characteristics — Does  the  type  of  program  (e.g.,  aviation,  missile,  IT,  etc.)  affect  how  long  a 
program  takes  and  how  well  a  program  schedule  can  be  predicted  and  maintained? 


10 


IDA 


Potential  Cycle  Time  Influence  Vectors 

Priority  and  focus 

-  How  important  are  schedules?  How  are  acquisition  schedules 
actually  set  and  how  are  they  changed?  What  are  these 
changes  based  on? 

Requirements  definition 

-  How  does  the  development  and  specification  of  requirements 
and  the  requirements  process  affect  program  schedules? 

Management  and  oversight 

-  How  does  program  management  incentivize  and  manage 
schedules?  How  do  oversight  processes  address  schedules? 

Program  definition  and  characteristics 

-  Which  types  of  systems  or  programs  demonstrate  which 
different  problems?  Do  different  types  of  systems  show  different 
outcomes?  Why? 


ll 


Intersects 


This  diagram  portrays  how  the  assessment  areas  relate  to  the  key  management  problems. 

As  will  be  elaborated  upon  later,  some  of  these  assessment  areas  seem  of  greater  importance  to  particular  management  problems, 
but  this  is  also  something  that  will  be  better  understood  as  research  proceeds.  The  large  Xs  in  the  slide  denote  the  areas  hypothesized 
to  be  of  greatest  relevance  for  future  research  focused  on  that  management  problem — for  example  in  “setting  schedules,”  “priority” 
and  “requirements”  are  likely  to  be  of  greatest  relevance,  while  “management  and  oversight”  is  less  of  a  factor  (denoted  by  a  smaller 
x).  Alternatively,  for  “schedule  growth”  our  hypothesis  is  that  “management  and  oversight”  will  be  the  dominant  factor. 

“Program  characteristics”  are  depicted  as  an  additional  dimension,  as  the  impact  of  different  types  of  systems  or  program 
characteristics  on  setting  schedules  and  managing  them  is  seen  as  variable  and  largely  undetermined — that  is,  certain  types  of  systems 
may  pose  much  larger  or  different  challenges  than  other  types.  However,  it  is  not  yet  clear  from  available  empirical  analyses  which 
programs  have  which  effects.  Thus,  while  program  characteristics  may  be  an  important  factor  in  reducing  cycle  times,  specifically 
which  types  of  programs  would  have  such  effects  is  an  open  empirical  question. 

Subsequent  slides  will  detail  IDA’s  initial  findings,  mostly  derived  from  the  detailed  literature  review,  on  each  of  these  research 
areas  and  then  recommend  additional  research  to  better  understand  the  area. 
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Initial  Findings:  Priority  and  Focus 


A  review  of  previous  research  indicates  that  schedules  for  defense  development  and  acquisition  programs  are  given  relatively 
little  focus,  in  relationship  to  other  concerns — especially  the  performance  of  the  system  and  its  cost.  Interviews  with  program 
managers  in  a  study  conducted  sixteen  years  ago  (McNutt,  1998)  found  that  schedules  were  generally  set  using,  at  best,  high-level 
milestones  and  timelines,  and  were  not  based  on  detailed  work  breakdown  analysis  that  specifically  considered  risks  of  achieving 
technical  objectives.  The  McNutt  study  made  an  important  observation:  the  schedules  set  early  in  the  program  development  process, 
although  rather  cursorily  derived,  become  the  basis  for  schedules  in  subsequent  requests  for  proposals  (RFP)  and  then  the  bids  by 
contractors.  Moreover,  if  schedules  are  given  this  rather  cavalier  treatment,  it  raises  questions  of  how  trade-offs  are  made  regarding 
what  capabilities  to  get  to  the  field  by  when.  Since  this  information  is  from  several  years  ago,  it  may  be  useful  to  follow  up  on  this 
topic  to  determine  whether  practices  have  changed. 
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IDA  Initial  Findings:  Priority  and  Focus 

•  IDA’s  review  of  prior  studies  strongly  indicates  that  cycle  time 
is  given  relatively  little  priority  and  defined  rather  arbitrarily  in 
project  planning,  management,  and  review.  Are  there  “levers” 
that  can  change  this? 

-  Establishing  an  analytical  basis  for  initial  operational  capability  (IOC)? 

-  Should  setting  the  IOC  wait  until  an  executable  (funded)  program  plan 
has  been  defined? 

•  IDA  identified  a  specific  Air  Force  effort  that  was  aimed 
explicitly  at  managing  cycle  time — What  were  its  results? 
What  happened  to  it? 

Lack  of  focus  and  rigor  in  setting  program  schedules  has 
consequences: 

•  Processes  don’t  adequately  address  the  value  of  getting  capabilities 
into  the  field  relative  to  costs  and  performance 

•  Unrealistic  schedules  become  embedded  into  the  requests  for 
proposals  (RFP)  and  contracts  with  subsequent  costs  and  delivery 
problems 
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Future  Research  Area  1:  Priorities 


Many  questions  can  be  raised  when  looking  at  how  program  baseline  schedules  are  determined  for  establishing  and  managing 
programs,  and  what  tools  and  data  program  offices  use  to  create  schedules. 

What  is  in  requirements  documents  regarding  when  a  capability  is  needed?  How  are  these  translated  into  program  schedules? 
How  are  these  schedules  reviewed  and  evaluated  relative  to  risks,  reasonableness,  and  trade-offs?  When  programs  are  modified,  how 
are  time  “needs”  considered? 

How  do  analysis  of  alternatives  (AOA)  treat  schedule?  (Corollary:  How  are  AOAs  treated?) 

What  are  the  practices  for  training  program  management  staff  in  schedule  setting?  Are  risk  analysis/risk  management  techniques 
applied  to  setting/assessing  schedules?  Are  these  used?  By  whom?  How  does  schedule  fit  into  the  systems  engineering  process  and 
what  is  the  schedule  based  upon? 

What  is  presented  to  defense  acquisition  boards  (DAB)  regarding  schedule  and  schedule  risk? — one  interesting  example  might  be 
the  Independent  Schedule  Assessment  input  to  the  F-22  aircraft  DAB. 

How  are  schedules  set  and  modified  in  the  contracting  process?  In  the  RFP,  how  are  schedules  presented  and  what  are  the 
rationales  underlying  them?  How  are  they  treated  in  the  source  selection  criteria  and  process?  How  are  schedule  incentives  provided  in 
contracts?  How  are  these  evaluated? 

Another  line  of  inquiry  is  to  explore  how  some  defense  program  organizations  have  effectively  determined  and  managed 
program  schedules.  Are  there  lessons  to  be  learned  and  practices  to  be  emulated?  For  example,  to  what  extent  are  program  schedules 
considered  in  the  annual  program  budget  decision  process  within  the  Components  and  OSD? 
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IDA  Future  Research  Area  1:  Priorities 

•  How  are  schedules  set?  Based  on  what  input  information, 
approaches,  tools,  decision  processes? 

-  Review  and  follow-up  on  Air  Force  study  (McNutt,  1998)  and 
subsequent  Air  Force  implementation  efforts 

-Access  and  review  scheduling  inputs  for  selected  MDAPs 

-  Identify  and  review  how  schedule  development  is  treated  in 
program  management  training  and  management  tools 

-  Interview  selected  SMEs  (e.g.,  former  PEOs/PMs) 

•  Identify  and  assess  examples  where  schedule-based  projects 
have  been  effectively  designed  and  managed. 

-  Review  literature  and  program  documentation — RAND,  NAVAIR 
(FI 8  E/F,  E-2D) 

-  Seek  to  identify  what  management  levers  are  used  by  “schedule 
disciplined”  programs  (e.g.,  are  there  explicit  contract  mechanisms 
used  in  the  R&D  contracts  to  spur/incentivize  meeting  schedules? 
How  are  these  drafted  and  overseen?) 
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Initial  Findings:  Requirements 


In  assessing  defense  acquisition  performance,  multiple  studies  have  identified  that  requirements  for  demanding  capabilities  are  a 
major  factor  in  why  major  defense  programs  take  a  long  time  to  achieve  their  results.  While,  perhaps  this  is  easy  to  understand,  it  is 
not  a  sufficient  explanation  for  defense  acquisition  schedules  for  several  reasons:  (1)  demanding  capabilities  have  been  achieved  in 
relatively  short  times — as  exemplified  by  the  F-l  17A  aircraft  which  met  an  IOC  of  four  years  (with  a  prior  HAVE  BLUE  prototype); 
(2)  estimating  in  the  face  of  demanding  capabilities  requires  understanding  and  evaluating  the  technical  status  of  the  technologies 
needed  for  the  proposed  capabilities  relative  to  when  they  can  reasonably  be  developed,  assessed,  and  integrated  into  a  defense  system 
and  it  is  unclear  whether  these  assessments  are  made  when  schedules  are  set;  (3)  demanding  capabilities  may  be  achieved  in  various 
manners,  such  as  through  incremental  or  spiral  approaches  or  through  separate  technology  development  efforts  parallel  to  systems 
development,  that  do  not  necessarily  involve  long  systems  cycle  times  to  deliver  some  level  of  capabilities  to  the  field — recognizing 
that  such  approaches  require  sound  cost-effectiveness  assessment. 

Highly  demanding  performance  becomes  “excessive”  performance  if  requirements  are  set  without  due  attention  to  and 
consideration  of  what  is  known  about  current  and  projected  technical  capabilities  and  the  uncertainties  in  achieving  the  stated  required 
level  relative  to  what  is  known  about  potential  adversary  capabilities.  Projecting  capability  requirements  and  the  rate  of  technology 
development  is  fraught  with  uncertainty,  and  the  farther  into  the  future  the  projection,  the  greater  the  uncertainty.  A  manager  seeking 
to  control  cycle  time  might  usefully  examine  how  projected  schedules  from  the  requirements  process  are  treated  and  incorporated  into 
the  systems  development  and  acquisition  process.  How  are  user  requirements  translated  into  project  development?  What  do  they  drive 
and  how  do  they  affect  schedules?  Does  the  requirements  process  contribute  to  efforts  to  implement  unproven  technologies?  What  can 
alleviate  this?  Do  requirements  processes  (AO As)  adequately  assess  or  trade  off  performance  against  time? 

Requirements  instability  has  been  shown  to  impact  cost  and  schedule  growth  in  defense  acquisition.  However,  there  are  several 
potential  reasons  for  requirements  to  change,  including  a  changed  view  of  the  national  security  threat  or  a  better  definition  of  what  is 
needed  to  meet  a  particular  threat.  Another  possibility  is  that  changes  are  made  to  provide  additional  or  different  (“improved”) 
capabilities  that  are  not  directly  justified  by  requirements,  but  pursued  through  an  agreement  at  the  program  management  and 
contractor  level.  Empirical  questions  are  (1)  How  much  have  requirements  changed  during  program  development?  (2)  What  were 
these  based  on?  (3)  How  did  they  affect  schedules?  (4)  To  what  extent  do  performance  “enhancements”  become  major  contributors  to 
schedule  growth  and  how  do  these  get  approved  in  the  program  management  process? 
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IDA  Initial  Findings:  Requirements 

There  are  two  “requirements”  problems:  excessive 
requirements  relative  to  the  state-of-the-art  and  changing 
needs  during  execution 

•  Requirements  processes  lead  to  “excessive”  performance/ 
technical  requirements  driving  excessive  times  to  IOC 

-  Linkage  between  user  requirements  and  project  schedules  is  not  well 
documented — what  do  they  drive  and  how  do  they  affect  schedules? 

•  Does  the  requirements  process  contribute  to  implementing  unproven  technologies  too 
quickly?  What  can  alleviate  this?  ICE? 

•  Do  AOAs  adequately  assess  or  trade  off  performance  against  time? 

•  Requirements  “instability”  has  been  shown  to  impact  cost  and 
schedule  growth  in  defense  acquisition — what  are  its  causes,  and 
how  have  some  programs  been  able  to  suppress  it? 

-  Do  requirements  processes  allow  for  or  accommodate  spiral  or 
incremental  approaches  to  provide  “needed”  capabilities?  To  what 
extent  have  such  approaches  been  successfully  implemented? 

-  “Lower-level”  performance  demands  inserted  into  the  development 
process  appear  to  stretch  schedules.  How  do  these  get  inserted  and 
how  are  they  assessed  relative  to  their  impacts  on  schedules? 
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Future  Research  Area  2:  Requirements 


A  basic  consideration  is  what  does  the  user  community  convey  to  the  development  community  concerning  when  the  capabilities 
it  wants  are  needed?  How  explicit  are  the  statements  of  when  the  capabilities  are  needed?  How  are  they  derived  and  what  are  they 
based  on?  How  much  flexibility  is  there  in  projected  times  and  what  are  the  risks  entailed  in  not  meeting  them? 

A  second  question  is  how  are  these  required  times-to-product  reflected  in  the  development  and  acquisition  program,  both  at  the 
program’s  formulation  and  in  its  execution?  When  it  becomes  apparent  that  the  program  will  not  be  able  to  deliver  the  specified 
capabilities  at  the  scheduled  delivery  milestone,  how  is  the  user  involved  in  determining  how  to  proceed?  How  is  the  effect  of 
stretching  the  scheduled  delivery  of  the  capability  evaluated? 

From  an  empirical  basis,  what  can  be  discerned  from  program  initiation  documents  and  subsequent  program  changes  on  how 
schedule  is  considered  in  the  execution  process?  When  program  schedules  are  changed  how  are  these  reflected  in  deliberations  with 
the  user  community? 

Schedule  has  been  proposed  as  a  key  performance  parameter  (KPP)  for  Major  Defense  Acquisition  Programs  (MDAP).  Has  this 
been  reflected  in  processes  to  set  these  schedules  and  assess  progress  based  on  them?  What  is  or  should  be  the  basis  of  a  schedule 
KPP? 

With  regard  to  requirements  instability — what  kind  of  requirements  are  changed  and  how  do  these  changes  affect  schedule? 
Where  in  the  process  are  these  changes  allowed  and  who  oversees  or  controls  their  execution?  Are  such  changes  frequently  done 
“informally”  and  thus  not  transparent?  If  the  scope  of  the  work  based  on  such  changes  increases  over  the  program’s  execution,  does 
this  impact  stretch  out  schedules?  How  is  this  reflected  in  management  concerns  about  time  to  product? 
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IDA  Future  Research  Area  2:  Requirements 

•  How  are  “user  requirements”  translated  into  program 
development  and  management? 

-  How  do  inputs  from  the  requirements  process  affect  the  product 
development  schedule? 

-To  what  extent  do  users  establish  the  intended  IOC  requirement 
that  becomes  part  of  the  initial  program  plan  as  reported  in  the 
Summary  Acquisition  Reports  (SAR)? 

-  How  are  the  impacts  of  schedule  changes  assessed  relative  to 
the  impact  on  requirements? 

-  How  are  changes  in  capabilities  during  development  assessed 
relative  to  schedule  impacts? 

•  How  are  the  operational  impacts  of  schedule  delay 
assessed? 

-What  analytical  tools  or  approaches  are  available  to  assess  such 
impacts  and  are  they  used?  By  whom?  Do  such  assessments 
impact  decisions? 

-  How  is  program  delay  communicated  from  the  program  to  the 
user  community  and  what  role  does  the  user  community  have  in 
schedule  delay  decisions? 
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Initial  Findings:  Management  and  Oversight 


Defense  development  and  acquisition  programs  are  complex,  highly  technical  endeavors  contracted  to  industrial  systems 
integrators  managed  through  Military  Service  program  offices.  Program  offices  exist  to  provide  capabilities  to  a  separate  user 
community,  but  work  within  a  changeable  financial  environment  that  affects  their  efforts.  Managing  such  programs  effectively 
requires  a  highly  skilled  and  trained  program  management  staff  having  the  tools  and  the  necessary  time  to  formulate,  contract, 
oversee,  assess,  and  provide  proper  guidance  for  their  programs.  Various  publications  and  review  groups  have  raised  concerns  about 
the  quality  and  sufficiency  of  these  staffs  as  well  as  the  analytical  tools  they  employ.  Another  concern  raised  is  the  lack  of  incentives 
for  these  staffs  to  address  schedules  and  to  raise  concerns  about  progress  on  achieving  results  within  specified  times.  Reportedly  there 
is  a  tendency  for  program  offices  to  try  to  work  with  the  contractors  to  adjust  or  fix  issues  of  performance,  which  often  entails 
stretching  out  schedules.  Yet,  there  are  examples  of  programs  that  have  delivered  complex  capabilities  on  schedule  and  within  cost. 
Are  there  lessons  to  be  learned  from  these  programs? 

A  second  area  of  concern  is  the  role  of  “oversight”  relative  to  program  management.  There  are  different  forms  of  oversight  at 
different  times  in  a  program  starting  with  its  inception.  Oversight  involves  several  different  organizations  and  responsibilities  at 
different  levels  from  the  program  office  to  the  service  acquisition  organizations  and  the  “corporate”  oversight  provided  by  the  Office 
of  Secretary  of  Defense.  While  some  have  claimed  that  oversight  is  itself  a  factor  in  causing  schedules  to  grow  (both  for  programs 
taken  together  over  time  and  within  individual  programs)  others  have  responded  that  the  oversight  process  (especially  test  and 
evaluation)  identifies  problems  that  should  have  been  addressed  earlier,  and  that  the  subsequent  stretch  of  schedules  is  due  to  earlier 
problems  that  the  management  and  oversight  process  did  not  identify  or  address.  What  recent  OT&E  review  (as  well  as  prior  research 
by  IDA)  shows  is  that  testing  reveals  problems  that  need  to  be  fixed  if  the  program  is  to  meet  its  performance  parameters,  but  these 
usually  require  additional  development  work  that  increases  both  the  system’s  schedule  and  cost.  What  is  not  well  documented  is  why 
such  problems  are  not  identified  or  addressed  before  IOT&E.  IDA  and  others  have  assessed  “bad  actors”  (that  is,  extreme  cases 
relative  to  cost  and  schedule  growth)  and  attributed  much  of  the  problem  to  poor  or  inadequate  focus  upfront  on  both  the  program  and 
system  design — which,  ironically,  would  require  greater  time  and  resources  in  the  pre-Engineering  and  Manufacturing  Development 
phase.  There  is  ample  evidence  that  inadequate  attention  to  such  analyses  and  engineering  upfront  subsequently  results  in  major 
program  difficulties. 
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IDA  Initial  Findings:  Management  and  Oversight 

•  Prior  studies  indicate  that  oversight  in  itself  is  not  a  major 
factor  in  cycle  time  (contrary  to  “conventional  wisdom”) 
Oversight  has  not  prevented  major  excesses,  such  as  F-22, 
Ground  Mobile  Radio  (GMR),  Future  Combat  Systems  (FCS), 
etc. 

-  Is  it  [1]  provided  at  the  wrong  time  on  the  wrong  things  with  too  little 
information  too  late  in  the  process?  [2]  ignored  or  overwhelmed  by 
other  factors? 

•  Management  incentives,  processes,  and  capabilities  are  seen 
as  inadequate  to  deal  with  issues  of  cycle  time.  Is  the 
oversight  process  not  adequately  identifying  problems  to  be 
managed? 

-  Incentives  for  addressing  problems  appear  to  put  schedule  last 

-  Incentives  for  raising  issues  are  weak 

-  Incentives,  in  general,  are  not  aligned  with  identifying  and  raising 
problems  (Who  loses  if  a  program  is  late?) 
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Future  Research  Area  3:  Management  and  Oversight 


There  has  been  relatively  little  research  on  how  program  management  applies  to  effectively  assessing  and  overseeing  schedules 
of  defense  programs.  Air  Force  research  (Wirthlin;  Air  Force  Institute  of  Technology  (AFIT);  McNutt)  explored  the  systems 
development  and  acquisition  process  focusing  on  what  drives  unfavorable  outcomes  in  terms  of  cost  and  time.  Some  factors  stand  out: 
the  process’  sheer  complexity,  the  number  of  parties  involved  with  different  motivations  and  agendas,  the  problem  of  time  to 
document  and  communicate  findings  and  positions  in  dispersed  operations.  More  elusive  is  understanding  how  some  programs  fare 
relatively  well  in  this  process  while  others  become  notable  for  excessively  long  times  to  fielding  relative  to  planned  schedules.  One 
possibility  is  that  aberrant  programs  are  in  fact  those  that  do  not  follow  the  strictures  of  the  management  processes  and  are  pushed 
through  despite  signs  of  incipient  problems — or  the  signs  are  not  identified  until  decisions  have  been  made  that  are  difficult  to  reverse. 
An  area  for  further  exploration  would  be  to  review  how  and  why  decisions  are  made  that  do  not  properly  assess  or  consider  the  risks 
of  unfavorable  outcomes  and  how  other  programs  have  mechanisms  that  adequately  address  such  factors.  Are  there  processes  and 
approaches  that  can  prevent  poor  outcomes,  or  are  these  largely  due  to  individual  decisions  to  proceed  despite  either  not  having  the 
information  or  not  properly  considering  it? 

What  management  factors  are  involved  in  poor  outcomes?  With  appropriate  management  capabilities  (including  requisite  skills, 
tools,  and  information)  exercised  could  many  problems  be  averted?  Alternatively,  are  extreme  schedule  and  cost  growth  more  often 
due  to  external  exigencies — such  as  budget  cuts,  high-level  changes  in  priorities,  or  changes  in  national  security  requirements?  If  so, 
internal  management  would  be  less  culpable  or  able  to  address  program  schedule  and  cost  impacts.  One  useful  exercise  might  be  to 
assess  extreme  outcomes  relative  to  whether  the  outcomes  were  primarily  due  to  the  unforeseen  and  uncontrollable  external  factors  or 
due  to  poorly  defined,  evaluated,  and  overseen  programs  where  problems  were  ignored  or  ineffectively  identified  or  addressed. 
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IDA  Future  Research  Area  3:  Management  and 

Oversight 

•  What  do  acquisition  process  assessments  show  as  factors 
leading  to  long  cycle  times  and  what  can  discipline  these? 

-  Review  AFIT  process  model  findings  to  identify  potential  control 
factors 

-  Review  and  assess  various  study  findings  indicating  that 
oversight  is  not  major  contributing  factor  (IDA,  OT&E,  McNutt) 

•  What  role  do  underlying  resource  factors,  such  as  training, 
skills,  tools,  insufficient  personnel  relative  to  tasks,  budget 
reductions  during  program  execution,  play  in  setting  and 
managing  program  schedules — how  are  these  allocated 
relative  to  other  demands? 

-  Review  prior  studies  (Defense  Science  Board  (DSB),  GAO, 
others?) 

-  Interview  IDA  SMEs  (former  PMs,  etc.) 

-  External  interviews  with  PMs  and  acquisition  executives 
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Initial  Findings:  Program  Characteristics 


Prior  research  has  shown  that  there  are  evident  differences  in  schedule  length  and  schedule  growth  for  different  types  of 
programs  at  different  times.  Analyses  have  shown  that  certain  types  of  weapon  systems,  such  as  combat  aircraft  and  satellites,  take 
longer  to  develop  and  acquire  than  other  systems.  Moreover,  some  types  of  systems  within  a  weapon  category  have  been  shown  to 
take  longer  than  others.  This  research  has  traced  some  of  these  results  to  underlying  factors,  such  as  complexity  or  the  introduction  of 
new  types  of  technologies  (such  as  digital  electronics  in  earlier  missile  systems).  What  is  apparent  from  these  analyses  is  that  not  all 
acquisition  programs  perform  badly — indeed  some  achieve  their  intended  results  close  to  the  initially  projected  times  and  costs.  While 
on  the  average  schedules  have  grown  some,  this  growth  (looking  at  medians)  may  not  be  a  major  concern,  given  the  additional 
complexities  of  current  systems  compared  to  those  of  the  past.  However,  it  is  also  the  case  that  throughout  defense  acquisition  there 
have  been  major  programs  that  have  taken  much  longer  than  others.  It  appears  that  these  aberrations  are  increasing  both  in  quantity 
and  in  terms  of  development  and  production  timelines.  Can  the  underlying  factors  of  these  extremes  be  isolated  relative  to  other 
programs  and  are  there  potential  means  to  identify  such  prospects  early-on  and  intercede  to  avoid  such  outcomes? 
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IDA  Initial  Findings:  Program  Characteristics 

Different  types  of  systems  have  shown  different  schedule 
behavior  over  time 

•  What  factors  contribute  to  the  differences?  What  do 
these  imply  about  understanding  schedules? 

•  How  to  measure  technological  maturity  and  system 
complexity  and  their  schedule  effects?  Have  any 
programs  been  specifically  managed  with  these 
concerns?  How  and  to  what  effect? 

•  What  characteristics  differentiate  “bad  actors”  from 
others?  Are  there  management  tools  for  identifying 
these  prospects  upfront  and  avoiding  them?  Are  there 
potential  l&W  factors  for  cycle  time? 

•  What  types  of  program  characteristics  are  related  to 
(drive)  schedule  problems,  e.g.,  concurrency, 
jointness? 
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Future  Research  Area  4:  Program  Definitions  and  Characteristics 


One  area  for  focus  is  to  look  at  how  “bad  actors”  have  affected  acquisition  system  aggregate  schedule  and  cycle  time  results. 
Large  differences  between  the  median  cycle  time  (such  as  time  to  IOC)  compared  to  the  mean  would  illuminate  the  effect  of  the 
outliers  on  the  acquisition  system’s  aggregate  performance.  This  could  be  assessed  for  different  types  of  systems  as  well  as  different 
time  periods. 

Another  avenue  of  research  is  to  determine  the  correlation  of  Technology  Readiness  Level  (TRL)  data  with  cycle  time/schedule 
data.  Do  low  TRLs  correspond  with  high/missed  schedules?  Are  appropriate  data  available  for  making  such  an  assessment? 

Selected  Acquisition  Report  (SAR)  data  augmented  by  data  from  the  requirements  process  could  be  used  to  assess  the  evolution 
of  the  planned  and  actual  schedule  over  time.  Parallel  data  for  the  KPPs  that  are  included  in  the  SAR  could  be  compared  to  these  dates 
to  see  if  there  is  a  relationship  between  schedule  and  performance.  Data  from  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR)  shows  the  evolution  of  procurement  plans  drawn  from  the  SARs.  Together,  these  data  might  support  analyses  of  how 
procurement  of  the  systems  was  affected  by  any  schedule  stretch  in  development. 

Another  assessment  could  use  historical  future  years  defense  plan/defense  program  projection  (FYDP/DPP)  data  to  give 
planned/actual  force  structures  which  would  show  when  equipment  is  fielded  as  well  as  how  equipment  would  be  retired  as  new 
equipment  comes  into  inventory.  Changes  in  these  data  might  show  how  delayed  procurements  affected  the  need  to  maintain  and 
support  older  systems. 

Another  focus  could  be  to  identify  measures  of  the  consequences  of  being  late  relative  to  other  impacts — such  as  costs  to  upgrade 
and  keep  legacy  systems  in  the  field.  One  clear  factor  is  the  cost  growth  of  the  systems  themselves  and  the  impact  of  the  reduced 
number  of  systems  fielded.  What  is  the  relationship  between  program  stretch  and  the  probability  of  a  program  being  cancelled  and  the 
attendant  impacts  of  funds  and  effort  wasted  as  well  as  unsatisfied  requirements?  Can  outcome  measures  be  derived  relative  to  force 
structure  impacts? 
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□A  Future  Research  Area  4:  Program  Definition  & 

Characteristics 

•  Which  types  of  systems  or  programs  demonstrate  cycle 
time  problems? 

-  Do  different  types  of  systems  show  different  outcomes?  Why? 

-  Differentiate  “drivers”  of  schedule  growth  for  development 
programs:  type  of  system,  complexity,  technical  maturity,  etc., 
and  assess  available  data  on  their  impacts  at  different  time 
periods. 

•  What  are  the  implications  of  the  distribution  of  current 
systems  developments  regarding  such  characteristics 
(type  of  systems  and  drivers  of  schedules)? 

-  What  analytical  tools  or  approaches  are  available  to  assess  their 
impacts? 

-  Can  these  be  used  to  direct  management  approaches  focused 
on  averting  potential  problems? 
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Initial  Findings:  Reducing  Cycle  Time 


A  recent  call  for  reduced  cycle  times  was  made  in  the  Office  of  Secretary  of  Defense’s  Better  Buying  Power  (BBP)  2.0,  which 
states  that  its  objective  is  to  “reduce  cycle  times  while  ensuring  sound  investment  decisions:  This  initiative  will  assess  the  root  causes 
for  long  product  cycle  times,  particularly  long  development  cycles,  with  the  goal  of  significantly  reducing  the  amount  of  time,  and 
therefore  cost,  it  takes  to  bring  a  product  from  concept  to  fielding”  (Kendall  2012). 

Embedded  in  this  objective  are  underlying  concerns  about  the  effectiveness  of  the  DOD  acquisition  system  to  deliver  needed 
capabilities  in  a  timely  manner.  These  concerns  reflect  a  number  of  prior  admonitions  from  a  range  of  organizations,  study  boards,  and 
reviews  that  the  DOD  acquisition  system  is  too  slow  in  meeting  defense  needs.  Periodically  there  have  been  calls  for  DOD  to 
fundamentally  alter  its  approach  to  acquiring  major  weapon  systems  with  the  objective  of  reducing  time-to-product.  Some  earlier 
assessments  have  called  for  reducing  cycle  times  by  as  much  as  50  percent.  The  BBP  2.0  statement  links  time  to  cost — indicating  that 
taking  longer  to  develop  military  capabilities  adds  to  their  costs. 

The  DOD  has  developed  and  built  extraordinarily  complex  systems  that  often  are  at  the  state  of  the  art  of  technological 
knowhow.  These  weapon  systems  and  even  more  complex  “systems  of  systems”  are  designed,  developed,  and  acquired  to  provide 
capabilities  to  assure  the  security  of  the  United  States  against  a  broad  range  of  current  and  future  threats.  The  United  States  military 
posture  stresses  that  its  military  capabilities  must  provide  technological  superiority  relative  to  its  potential  adversaries.  Because  of 
these  demands,  DOD  weapon  systems  are  costly  and  considerable  time  is  needed  to  develop  and  produce  systems  of  their  scale,  scope, 
and  complexity.  However,  the  concern  is  that  these  trends  have  reached  a  point  where  the  cost  and  time  are  becoming  too  great 
relative  to  getting  the  needed  capabilities  into  the  field.  In  a  world  of  globalized  technology  where  potential  adversaries  and 
competitors  are  more  rapidly  accessing  and  developing  advanced  technological  capabilities,  time-to-product  may  be  much  more 
important  than  in  the  past.  Can  defense  acquisition  cycle  times  be  substantially  reduced  and  still  achieve  needed  defense  capabilities? 

It  is  important  to  emphasize,  however,  that  reducing  cycle  times  in  any  substantial  way  raises  fundamental  questions  about 
Defense  Acquisition  Strategy  (what  is  to  be  bought  in  what  way)  that  are  beyond  acquisition  management  and  oversight.  It  is  likely 
that  to  truly  reduce  acquisition  cycle  times  on  the  order  of  50  percent  would  entail  a  revamped  acquisition  approach  using  a  strategy  of 
strict  adherence  to  proven  technologies  in  a  spiral  development  process.  Indeed,  this  is  precisely  the  approach  taken  by  leading  high 
technology  systems  firms.  While  such  approaches  have  been  advocated  by  some  for  DOD,  they  also  are  not  currently  embedded  in  the 
DOD  systems  development  and  acquisition  process  (or  represented  by  DODI  5000.02). 


30 


IDA  Initial  Findings:  Reducing  Cycle  Time 

•  The  Office  of  Secretary  of  Defense’s  Better  Buying  Power 
2.0,  states  that  its  objective  is  to: 

Reduce  cycle  times  while  ensuring  sound  investment  decisions: 
This  initiative  will  assess  the  root  causes  for  long  product  cycle 
times,  particularly  long  development  cycles,  with  the  goal  of 
significantly  reducing  the  amount  of  time,  and  therefore  cost,  it 
takes  to  bring  a  product  from  concept  to  fielding. 

•  Studies  refer  to  commercial  cycle  time  as  an  objective  for 
DOD — citing  “time-to-product”  as  key  business  driver 

•  Defense  systems  may  have  characteristics  (complexity, 
use  in  extreme  environments,  etc.)  that  prevent  strict  use 
of  commercial  practices,  but  some  lessons  may  transfer 

•  Competitors  and  adversaries  are  more  quickly  adopting 
advanced  capabilities,  raising  concerns  about  U.S. 
technological  superiority 

Reducing  cycle  times  raises  fundamental  questions  regarding 
Defense  Acquisition  Strategy  (what  is  to  be  bought  in  what  way) 
that  are  beyond  management  and  oversight 
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Future  Research  Area  5:  Approaches  to  Reduce  Cycle  Time 


From  the  perspective  of  identifying  further  analyses  to  achieve  better  managed  programs,  one  focus  might  be  on  developing 
ways  to  institute  effective  scheduling  as  a  management  practice  within  the  Department  of  Defense. 

To  what  extent  and  to  what  effect  have  there  been  specific  attempts  to  manage  or  reduce  defense  program  schedules?  What  has 
been  attempted  and  what  have  been  the  results? 

How  do  different  acquisition  strategy  or  approaches  (for  example,  spiral  acquisition  and  agile  development)  address  schedules; 
have  these  been  employed  in  defense  and  with  what  results? 

This  raises  two  different  management  questions:  (1)  how  to  manage  and  discipline  schedules  under  the  current  acquisition 
process  and  (2)  how  to  reduce  acquisition  cycle  time  using  fundamentally  different  acquisition  processes.  This  distinction  is  important 
because  it  sets  some  boundaries  on  what  is  achievable  relative  to  time,  cost,  and  probability  of  success.  What  strategies  are  likely  to 
succeed?  Have  any  such  attempts  been  made  and  have  they  succeeded  (Younossi,  et  al.,  2005)? 

Finally,  reducing  acquisition  cycle  time  should  be  assessed  and  related  to  broader  management  objectives.  It  is  generally 
accepted,  and  recent  studies  verify,  that  program  cycle  times  should  be  less  if  the  program  is  based  on  less  advanced  and,  thus,  better 
known  technologies  and  processes.  A  crucial  element  of  determining  schedules  is  what  is  trying  to  be  achieved — and  this  directly 
relates  to  requirements  and  the  technology  development  process.  Given  the  uncertainties,  both  in  the  requirements  and  the  funding 
processes,  shouldn’t  the  focus  be  on  greater  agility  and  flexibility  in  the  acquisition  system?  Such  agile,  flexible,  adaptable  approaches 
have  explicit  implications  for  setting  schedules  and  changing  them.  Can  such  processes  be  implemented  effectively  in  the  DOD  (Patel 
and  Fischerkeller  2013)? 
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.  Future  Research  Area  5:  Approaches  to 

Reduce  Cycle  Time 

Priority  and  focus:  What  explicit  attempts  have  been  made  to  manage 
and/or  reduce  cycle  times  with  what  results? 

Requirements  definition:  How  do  warfighter  needs  from  field  (e.g., 
joint  urgent  operations  needs  (JUON)  affect  program  schedules? 

Management  and  oversight:  Are  there  management  tools  and 
approaches  that  can  be  used  to  dramatically  reduce  cycle  times? 
When  should  these  be  used? 

Program  definition  and  characteristics:  Which  types  of  systems  or 
programs  can  be  managed  to  significantly  reduce  cycle  times?  What 
are  their  characteristics? 
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Some  Broader  Concerns 


The  purpose  of  this  initial  research  and  the  future  program  of  research  proposed  in  this  paper  is  to  provide  a  basis  for  better 
understanding  defense  acquisition  program  cycle  times — the  time  it  takes  to  get  from  initial  concept  definition,  through  development 
and  production  into  an  operating  capability.  This  research  program  is  motivated  by  a  long-standing  concern  in  the  DOD  that 
acquisition  program  cycle  times  are  too  long  and  getting  longer,  but  is  focused  on  more  specific  questions: 

Data  show  that  cycle  time  distributions  (similar  to  cost)  are  highly  skewed  with  several  aberrations  of  extremely  long  cycle  times 
as  large  outliers  from  the  norm.  From  a  management  perspective  should  identifying  and  reducing  such  outliers  be  the  most  important 
aspect  of  cycle  time  management?  Should  the  understanding  of  weapon  system  cycle  time  be  focused  on  the  extremes  of  the 
distribution?  What  are  the  implications  of  such  extremes  relative  to  “normal”  acquisitions? 

Is  the  notion  of  “should  time”  (analogous  to  “should  cost”)  appropriate?  If  so,  how  should  it  be  derived?  Is  the  idea  that  cycle 
time  either  in  general  or  on  average  should  be  reduced  by  some  factor  (e.g.,  50  percent  or  25  percent)  based  on  a  sound  rationale? 
Does  it  over  simplify  the  problem  by  ignoring  the  interplay  between  time,  cost,  and  performance?  Does  it  appropriately  consider  the 
complex  nature  of  defense  systems?  Data  show  that  the  “norm”  (i.e.,  median)  cycle  time  for  systems  has  generally  increased  over 
time.  Should  this,  in  itself,  be  a  significant  management  concern? 

How  important  is  the  content  of  the  cycle  rather  than  the  overall  time?  Are  certain  elements  taking  longer;  if  so,  why?  Should 
some  take  longer  to  improve  overall  results? 

Generally,  schedule  estimates  have  been  overly  optimistic,  with  actual  times  longer  than  those  estimated.  Should  estimates  of 
cycle  time  be  developed  differently?  Are  these  estimates  a  management  concern? 

How  should  cycle  time  be  assessed  relative  to  program  failures,  where  programs  were  cancelled  either  because  their  cycle  times 
were  so  out  of  control  or  the  threat  for  which  they  were  designed  was  judged  to  have  changed?  Is  there  a  relationship  between 
schedule  problems  and  prospects  of  program  failure? 

Is  cycle  time  an  indicator  of  the  need  for  a  different  way  to  acquire  defense  capabilities?  If  the  cycle  times  in  the  current 
approach  are  seen  as  too  long  and  getting  longer,  does  this  indicate  the  need  to  develop  and  acquire  capabilities  differently? 
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Some  Broader  Concerns 


IDA 


•  Is  the  notion  of  “should  time”  appropriate?  If  so,  how 
should  it  be  derived? 

•  From  a  management  perspective,  should  identifying  and 
reducing  extreme  outliers  be  the  most  important  aspect 
of  cycle  time? 

•  How  important  is  the  content  of  the  cycle  phases  rather 
than  the  overall  time? 

•  Is  there  a  relationship  between  cycle  time  problems  and 
prospects  of  program  failure? 

•  Does  a  longer  cycle  time  indicate  the  need  for  a  different 
way  to  acquire  defense  capabilities? 
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Preliminary  Work  Plan 


Periodically,  defense  weapon  system  acquisition  cycle  time  (i.e.,  the  time  it  takes  to  get  a  weapon  system  program  through 
development  into  production  and  fielding)  has  been  flagged  as  a  concern.  While  there  is  some  evidence  that  cycle  time  has  gotten 
longer  in  recent  decades,  there  is  no  clear  evidence  that  it  is,  in  itself,  a  major  problem  or  is  “getting  worse,”  since  it  could  be  more  a 
result  of  the  increased  complexity  of  the  systems  rather  than  degraded  organizational  performance.  It  is  also  not  clear  how  much  the 
general  (average)  increase  in  cycle  time  affects  operational  capabilities.  There  is,  however,  evidence  of  a  problem  with  how  schedules 
are  derived  and  how  they  are  addressed  in  the  DOD  development  and  acquisition  process.  These  problems  include: 

•  Many  systems  coming  in  later  than  projected,  raising  questions  regarding  how  initial  schedules  are  set  as  well  as  downstream 
implications  of  such  underestimation 

•  Highly  skewed  variation,  with  outliers  indicating  prospects  of  major  problems  for  certain  types  of  systems  or  approaches  that 
result  in  extraordinary  or  aberrant  delays  that  may  have  major  implications  for  defense  capabilities 

•  Pursuit  of  capabilities  that  are  well  beyond  the  current  state-of-the-art  (and  the  impact  on  schedule  and  cost)  relative  to  less 
ambitious  capabilities  that  could  be  delivered  sooner  (which  raises  the  questions  of  how  trade  off  of  quicker  schedules  relative 
to  greater  capabilities  should  be  done  (Francis  2013)) 

Based  on  these  initial  findings,  IDA  concludes  that  further  research  is  needed  on  the  areas  of  assessment  presented  in  this 
briefing. 


While  a  system  being  “later”  than  initially  projected  most  likely  does  mean  that  it  will  be  more  costly  (per  unit)  and  the  overall  program  costs  will  rise,  it  is 
unclear  under  what  circumstances  the  system  earlier  would  provide  more  beneficial  capabilities.  For  example,  a  system  might  be  late  because  its  technical 
risk  was  underestimated  and  its  development  had  to  be  stretched.  The  implication  is  that  for  the  given  capability  sought  (“required”)  more  technology 
maturation  was  required  which  would  have  taken  more  time.  Thus  it  would  not  be  the  case  that  the  system  [as  defined]  would  have  gotten  to  the  field  sooner, 
or  even  cheaper).  Perhaps  the  relevant,  but  more  difficult,  question  is:  When  is  a  system  too  late  relative  to  stated  needs,  such  that  it  is  no  longer  useful  or 
effective,  and  what  are  the  implications  of  this? 
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